Everything about The Principle Of Maximum Entropy totally explained
The
principle of maximum entropy is a method for analyzing available qualitative
information in order to determine a unique
epistemic probability distribution. It states that the
least biased distribution that encodes certain given information is that which maximizes the
information entropy.
The principle was first expounded by
E.T. Jaynes in
1957 when he introduced what is now known as
Maximum entropy thermodynamics: an interpretation of the
Gibbs algorithm of
statistical mechanics. He suggested that
thermodynamics, and in particular thermodynamic
entropy, should be seen just as a particular application of a general tool of
inference and information theory.
The maximum entropy principle is like other
Bayesian methods in that it makes explicit use of
prior information. This is an alternative to the methods of inference of classical statistics.
Testable information
The principle of maximum entropy is useful only when applied to
testable information. A piece of information is testable if it can be determined whether a given distribution is consistent with it. For example, the statements
» The expectation of the variable
x is 2.87
and
» p2 +
p3 > 0.6
are statements of testable information.
Given testable information, the maximum entropy procedure consists of seeking the
probability distribution which maximizes
information entropy, subject to the constraints of the information. This constrained optimization problem is typically solved using the method of
Lagrange multipliers.
Entropy maximization with no testable information takes place under a single constraint: the sum of the probabilities must be one. Under this constraint, the maximum entropy probability distribution is the
uniform distribution,
»
All that remains for our protagonist to do is to maximize entropy under the constraints of her testable information. She has found that the maximum entropy distribution is the most probable of all "fair" random epistemic distributions, in the limit as the probability levels go from discrete to continuous.
Compatibility with Bayes Rule
Recently, it has been shown that
Bayes' Rule and the Principle of Maximum Entropy (MaxEnt) are completely compatible and can be seen as special cases of the Method of Maximum (relative) Entropy (Giffin 2007). This method reproduces every aspect of orthodox Bayesian inference methods. In addition this new method opens the door to tackling problems that couldn't be addressed by either the MaxEnt or orthodox Bayesian methods individually.
Further Information
Get more info on 'Principle Of Maximum Entropy'.
|
External Link Exchanges
Do you know how hard it is to get a link from a large encyclopaedia? Well we're different and will prove it. To get a link from us just add the following HTML to your site on a relevant page:
<a href="http://principle_of_maximum_entropy.totallyexplained.com">Principle of maximum entropy Totally Explained</a>
Then simply click through this link from your web page. Our crawlers will verify your link, extract the title of your web page and instantly add a link back to it. If you like you can remove the words Totally Explained and embed the link in article text.
As long as your link remains in place, we'll keep our link to you right here. Please play fair - our crawlers are watching. Your site must be closely related to this one's topic. Any kind of spamming, dubious practises or removing the link will result in your link from us being dropped and, potentially, your whole site being banned. |
